DesignCon 2004 A High-Channel-Density, Ultra- High Bandwidth Reference Backplane Designed and Manufactured for 10 Gb/s NRZ Serial Signaling

In the last 12 months 10Gb/s serial data transmission in copper backplane has moved from being a good possibility to a practical reality. Advancements both in active signal conditioning and passive interconnect technologies have made this happen. Today it is feasible to build, in volume production, a long reach multiple channel copper backplane that can deliver an aggregate bandwidth of hundreds or thousands of Gb/s over copper. The doubts and concerns regarding the manufacturability of such a backplane have been erased through hard evidence. This paper presents one such evidence. Of course it is not an easy task to design such a high-bandwidth mass-producible backplane—there is a significant learning curve associated with understanding the new technologies as well as the necessary trade-offs between them. While many system vendors are embracing the new technologies and embarking on high-end backplane designs, others are not because of both the fear of the unknown and the recent downturn of the economy. To aid in broadening the understanding of what it takes to implement a fully functioning high-density switch fabric that can deliver hundreds of Gb/s in a mass-producible configuration, this paper provides a reference design for a high-performance reference backplane capable of 10Gb/s serial NRZ signaling over copper, over multiple channels, and over long reach. It addresses the key challenges and describes design features that are important for 10Gb/s serial applications. It emphasizes each phase of a backplane design—architecture definition, modeling, simulation, measurement and manufacturability. It also describes how to establish the right combination of active signal conditioning and passive connectivity technologies to achieve repeatable 10Gb/s serial data transmission in high channel-count mass-producible applications. The backplane system has been designed and developed jointly by Winchester Interconnect Technologies and Xilinx. It is representative of a typical telecommunication or data communication system application in terms of configuration, routing density, transmission length, routing complexity, size and manufacturability. The daughter cards use the Xilinx® RocketIOTM X embedded transceivers, which are capable of driving optics as well. The passive channel, including backplane, connectors and daughter card transmission line structures, is based on Winchester’s SIP1000 I-Platform connector and printed circuit board technology. The backplane and the daughter cards are built and assembled at the high volume production unit of Winchester Interconnect Technology. Authors Biographies John Mitchell, Sr. Application Engineer, Winchester Electronics/Interconnect Technologies Mr Mitchell has 20 years experience in the field of computer and network equipment design. He spent much of his career at Unisys developing mainframe and large server systems. Most recently before joining Winchester he was Director of Hardware Engineering at Optical Networking start-up LuxN. He received his BSME from Purdue University. Bodhi Das, Signal Integrity Specialist and Program Manager, Xilinx, Inc. Mr. Das works for the Communications Technology Division of Xilinx at the San Jose (California, USA) office. His responsibility includes managing engineering programs, signal integrity engineering, applications, technical marketing, technology partnerships, and industry standards activities. He has a Bachelors degree from Indian Institute of Technology, Kharagpur, India, and a Masters degree from Iowa State University, Ames, Iowa, USA, both in Electrical Engineering. Bodhi has numerous journal and conference publications, and 2 US patents. Introduction Over the last year, a number of companies have introduced products and technologies in the fields of semiconductor, connector and printed circuit board (PCB) design that can enable 10Gb/s serial signaling in a system backplane environment. Still, most system designers have been reluctant to embrace these new technologies and hesitant to design new higher speed systems that can drive 10Gb/s serial signals over a backplane – uncertain that practical systems can really be made with these new components. The objective of this paper is to demonstrate and document that 10G serial backplane systems, based on mainstream non-return-to-zero (NRZ) signaling, can be designed and manufactured today to operate error-free in a typical application system featuring multi channel crosstalk. The paper describes the system attributes of a physical layer reference design that was built using active devices from Xilinx and passive channels from Winchester and Interconnect Technologies , and reviews the key enabling technologies used for successful implementation of 10Gb/s serial systems. The design process and the importance of modeling and simulation are discussed. Finally, test results for passive and active interconnect systems are presented. System Attributes The physical dimensions and configurations of the design are representative of typical applications where these technologies will be employed. A 19” rack mount Advanced TCA Architecture was chosen as the form factor of the system. This system architecture is being widely accepted by numerous OEM’s as the basis of new designs. It is also representative in terms of physical layout to many proprietary system designs. Note: For this reference design, we are leveraging only the system form factors and not the management, control plane signaling and other aspects of the specification; the Advanced TCA 2.5Gb/s serial data plane signaling is replaced with 10Gb/s serial signaling. Figure 1 shows the general arrangement and form factor of the Advanced TCA architecture. Figure 1. Advanced TCA, Chosen as Form Factor for Reference Design The design implements 14 line cards plugging in to a backplane on 1.2” slot centers. Typical applications have backplane connectivity in the form of star or mesh architecture. Mesh architectures are usually more difficult to route and require the most routing layers. The backplane and daughtercard routing layers have been sized to support a typical full mesh architecture. The system is designed to examine several channel lengths up to 1 meter. Previous studies have shown differences in thru channel performance depending on the board layers that were used in routing the channel. In this design the effect of crosstalk on the routing layers used will also be demonstrated. The system is designed to isolate and characterize top, middle and bottom layer routing. Key Technologies Employed A number of key technologies have come together to enable this practical 10Gb/s serial NRZ system to be possible. Connector Technology The Winchester SIP1000 connector, shown in Figure 2, was specifically designed to meet the demands of 10Gb/s+ data transmission in a backplane environment. Its features include: • Low loss (<-0.5dB at 10Gb/s) and low crosstalk (<1% at 35ps rise time) in connector • Single-piece compression-mount architecture which eliminates need for impedance distorting pin-in-hole terminations and facilitates the use of small via technology • Full flexibility in backdrilling to tune vias for impedance • Footprint optimized for low crosstalk in vias to minimize overall channel crosstalk (see Figure 3) • Side-by-side differential pair configuration eliminates skew in channels • High Density 28 diff pair per inch capacity Figure 2. SIP1000 Connector Figure 3. SIP1000 Footprint PCB Technology Boards for this system are manufactured by Northrop Grumman Interconnect Technologies’ high volume production PCB facility. The facility is registered ISO 9002 and has long been recognized as a world leader for backplanes and high speed printed circuit boards. Three key PCB technologies are employed to optimize performance for 10Gb/s serial data transmission. Low loss and minimum loss dispersion are achieved by using low loss material. In this case we have chosen Rogers 4350 material. The dramatic performance improvement achieved by using low loss material is illustrated here. Channel pulse response to a 1V input pulse is shown for Rogers 4350 and Nelco N4000-13 for a 34” channel length in Figure 4. Many PCB manufactures are now processing Rogers and other low loss materials with good yields. The nominal additional cost of low loss material is highly justifiable and is easily offset by simpler silicon and lower system power. In any case, the cost of a finished backplane is dominated by the processing costs with the material choice having a secondary cost impact. Figure 4. Pulse Response of Low Loss Channels Via design and implementation is the second critical PCB technology that is required for 10Gb/s systems. Two key aspects of via design are tuning of the via for the correct impedance, and eliminating the stub formed when the signal path does not continue to the end of the via. Via tuning is achieved by adjusting via size, pad size and anti-pad size to match the impedance of the via to the impedance of the transmission line. Using small vias allows excellent via impedance matching while still maintaining reference planes for transmission lines that route thru the connector field. For 10Gb/s speeds, it is critical to also eliminate the stub of unused via length. The most practical and cost effective way to do this is with a process known as back-drilling or counter-boring. Back-drilling is a secondary drill process that is performed after the initial via is drilled and plated. Back-drilling is relatively new to the digital world, but not to the RF world. Interconnect Technologies has been building production boards with this process for over 8 years with excellent yields and relatively little additional cost. Figure 5 shows pictures of both small via diameter and back-drilling from test coupon on the reference system backplane. Figure 5. Key via technologies include small diameter vias (left) and via backdrilling (right). (Pictures are from test coupon on reference system backplane.) The third key PCB technology required for 10G design is tight process control and testing capability. It is not enough to know the nominal performance of a high speed channel, it is critical to know the overall variation in channel performance that the manufacturing process can support over a large sample of assemblies. The simple transmission line section shown in Figure 6 can give rise to variations in line/space width, copper thickness, copper surface roughness, dielectric thickness, dielectric properties. These variations require understanding and control. They also need to be accurately verified in the manufacturing process. Figure 7 shows a sample of insertion loss S-parameters for a sample of nominally identical transmission lines. Figure 6. Transmission Line Figure 7. Variation in S-parameter Performance An example of backdrilling to eliminate the stub is shown here. Example of small via that enables full impedance tuning. (Drill hole aspect ratio is 16:1) Silicon Technology The transceiver used for this reference design is the embedded RocketIO X transceiver in Xilinx® Virtex-II Pro X FPGA. Virtex-II Pro X, like other Xilinx FPGAs, have a wide range of applications from telecom, datacom, storage, computer systems to defense, medical appliances, test equipments, instrumentation, electrical appliances, etc. Virtex-II Pro X is the first FPGA in the industry that embeds a 10Gb/s transceiver. Key silicon technologies for 10G serial backplane channels are transmit preemphasis and receive post-equalization. The use of 10Gb/s capability in the Virtex-II Pro X is significant, in that FPGAs are the most widely used prototyping vehicles for new design launches. Transmit Pre-Emphasis and Receive Post-Equalization As data rates increase, bandwidth limitations in the channel produce an increased amount of deterministic jitter (DJ), more specifically known as Intersymbol Interference (ISI). Emphasis / equalization allows the user to offset these bandwidth limitations by boosting higher frequency components within the serial data. The transmit pre-emphasis and the receive post-equalization essentially accomplish a similar function, but they are implemented differently. On the transmit side, emphasis boosts the high frequency component and attenuates the low frequency component of the transmitted signal out of the line driver but before the IC package. On the receive side, equalization performs the same operation on the received signal after it passes through the IC package. Both the TX emphasis and RX equalization are programmable to different states so that optimum signal compensation can be achieved through appropriate programming. The line driver design for the transmitter of the RocketIO X transceiver is based upon Current Mode Logic (CML) techniques. This driver is terminated with 50 Ohms (100 Ohms differential) to the power supply. It can be connected to the receiver using AC or DC coupling. Pre-emphasis is built into the design of the CML driver, whereas post-equalization is a part of the receiver analog front end (RX AFE). Features of the Xilinx Virtex-II Pro X include: • Variable speed full-duplex RocketIO X transceiver, allowing 2.488 Gb/s to 10.3125 Gb/s data rates. This includes specific baud rates used by various standards such as SONET OC-48 and OC-192, PCI Express, Infiniband, XAUI (10G Ethernet and 10G Fiber Channel), XFP, Xilinx Aurora and upcoming 10Gb/s standards such as Common Electrical Interface (CEI) of Optical Internetworking Forum (OIF) and UXPiTM. • Depending on the device, a Virtex-II Pro X FPGA has between 8 and 20 transceiver modules. • Monolithic clock synthesis and clock recovery system • Automatic lock-to-reference function • Serial output differential swing can be programmed between 200 mV to 1600 mV (peak-peak), allowing compatibility with other serial system voltage levels. • A wide range of discrete levels of programmable transmit pre-emphasis. • Linear equalizer in receiver, fully programmable for rectifying various amounts of channel losses • AC and DC coupling • On-chip termination of 50 ohms eliminating the need for external termination resistors • Pre and post driver serial and parallel TX to RX internal loopback modes for testing operability • Programmable comma detection to allow for any protocol and detection of any 10-bit character • 8B/10B and 64B/66B encoding blocks • Full-function FPGA with embedded PowerPC processor Design Process and the Importance of Modeling and Simulation Critical to successful design of 10G systems is a structured design process with emphasis on modeling and simulation. The detailed design process for this reference design is as illustrated in Figure 8. Figure 8. Design Flow 1. Stack Configuration The process starts with an estimate of the routing layers that will be required for the implementation of the backplane interconnect architecture desired. In this case, the ultimate goal is to implement a full mesh architecture. To accommodate this 30 layers have been allocated for the backplane (14 signal layers) and 16 layers for the daughtercards (7 signal layers). 2. Channel Component Modeling Channel component modeling begins with the transmission line. A proper model requires accurate material properties and proper modeling techniques. Appropriate behavior of the model in time domain simulations requires additional care. Connectors and vias typically require 3-D field solvers to accurately model performance. Models are constructed to extract both thru and crosstalk performance. Figure 9 shows an Ansoft HFSS model that combines the connector, the interface and the PCB vias into a single model. The model’s ports are defined at uniform transmission line sections. Figure 9. 12 Port Via Model for 30 Layer PCB and Connector Interface 3. Channel Modeling Individual component models are then linked together to run end-to-end channel simulations. The result is an equivalent channel model that can be used to export S-parameter models for both thru and crosstalk channels. The channel model in Figure 10 includes one thru channel and 2 crosstalk channels that traverse from a daughtercard on one end of the backplane to a second daughtercard on the far end of the backplane. Figure 11 shows the exported S-parameters for this modeled channel. Figure 10. Ansoft Designer Model linking individual component models together to create composite channel model. Connector & 2 Vias Connector & 2 Vias Backplane 12.8” T-line Daughtercard 6.4” T-line Daughtercard 6.4” T-line Diff Port 1 Diff Port 2 Figure 11. Insertion and reflection S-parameters of composite channel With S-parameter channel models that have either been modeled as described above or measured directly from test structures, active channel simulations can be performed. Simulations are performed with ideal drivers as well as drivers with jitter added. Simulations with different signal conditioning approaches are run. Figure 12 shows simulation of a driver with jitter, combined with a channel with crosstalk, and a receiver with linear equalization. Figure 12. Active channel simulation with jitter, crosstalk and linear equalization. Taking the simulations one step further, probability density functions for the driver performance, data pattern, and receiver performance can be combined with channel models to predict active channel Bit Error Rates. 4. Design Rule Setting Channel modeling and simulation allow us to optimize the system for both thru and crosstalk channel performance. From this effort a set of design rules for both the backplane and daughtercard are established. Figure 13 summarizes the established constraints. Feature (units mils) Backplane Daughtercard Line width/space 6.25/9.25 6.25/9.25 Pair-to-pair space 58 58 Via diameter (drill) 13.5 13.5 Board thickness 220 116 Anti Pad sizes 48-54 44-56 Figure 13. Design rules for reference design 5. Layout With design rules established, layout of the backplane and daughtercard can be completed. Routing of the reference design includes 9 channel types as detailed in Figure 14. Figure 14. Channel types in reference design Each channel type includes the thru (victim) and 8 significant crosstalk (aggressor) channels. A section of the backplane layout is shown in Figure 15. Middle 40” 8 40” 40” 20” 20” 20” 8” 8” 8” Channel Length